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. statement of Focus 




The proposed taxonomy of programmatic tasks In an educational 
evaluation and facilitation coordination system encompasses 
tasks appropriate for evaluators, members of an evaluation 
unit, and members of facilitation and coordination units. 

In this comprehensive approach to the evaluation process, the 
taxonomy classifies and specifies tasks to effect evaluation, 
facilitation and coordination. 

Such a conceptualization of the evaluation process differs 
from that of Owens and Stufflebeam. In our view, the process 
becomes more rigorous and necessitates the Identification and 
performance of additional tasks. For example, the development 
of managerial plans. 
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1. DEVELOPING SUPPORTIVE CLIMATES FOR EVALUATION 

There ere two major conceptions of evaluation! (1) the traditional 
Givaluation approach which focuses upon marking , grading » crediting, 
and the attainment of behavioral objectives through specific transac- 
tions; and (2) the more modern or hebteric evaluation approach which is 
intent on providing the decision-maker with the kinds of information he 
requires in order that he may arrive at a "rational" decision about the 
"real" world. In the neoteric evaluation approach, "Anything goes... any 
reasonable means to an end." 

Despite this amazing trend in evaluation theory, evaluation specialists 
and select members of the educational community (EM) have not appreciably 
improved their working relationships that traditionally have jeopardized 
whatever mutual efforts have been undertaken. Since most educators are 
required to grade their students, they will evaluate or measure student 
outcomes. In general, however, educators resist or reject the use of 
evaluation for other purposes. They--or so it seems— have no desire to 
initiate or to participate in either traditional or neoteric formal 
evaluation efforts. 

Many evaluators have arbitrarily assumed that the educators* professional 
insecurity generates this uncooperative attitude. However, some 
evaluators* insensate approach could catalyze negative reactions. In 
actuality, the educators* disillusionment with traditional evaluation 
practices may well account for such negativism and hostility. 

For example, Stake^ has suggested that '*Today*s educator may rely little 
on formal evaluation because its answers have seldom been answers to 
questions he is asking." (p. 523) Egon Guba^ concurs: "The traditional 

methods of evaluation have failed educators in their attempts to assess 
the impact of innovations in operating systems." (p. 29) 

One must accept the implications that Stake and Guba draw: evaluators and 
evaluation efforts are resisted by the educational community because the 
"pay-off" they have received to date has been totally inadequate. Indeed 
Guba^ argues that traditional evaluation efforts have hurt more than 
helped the educator. 

Efforts to relieve this impossible situation by creating neoteric 
evaluation models which are more acceptable to EM members have met, so 
far, with only partial success. Even reasonable expectations that 
recently— enacted legislation would rekindle the educators latent 
interest in evaluation (traditional or neoteric) have been extinguished— 
predicted demand situations simply have not materialized. It would seem 
that most EM members either do not know about the dramatic changes which 
have occurred in evaluation strategies or do not fully comprehend the 
meaning of such changes. 



^Stake, R.E., "The Countenance of Educational Evaluation." 
Teachers College Record, 1967, 68, p. 523 - 540 

2cuba, E.G., "The Failure of Educational Evaluation." 
Educational Technology, 1969, IX, p. 29 - 38 
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This communications problem must be solved: EEFCS participants have the 
obligation to Inform the educational community of the many advances In 
evaluation strategies and techniques. Once the educational community 
realizes the need for evaluation, transition to a demand situation 
(though still great) Is no longer that great. 

Performance of the following tasks would Induce a supportive climate for 
evaluation: 

1» 1 Developing alternative strategies based upon particular 
characteristics of select members or segments of the 
educational milieu In order to modify reactions towards 
evaluation 

1» 2 Assessing the existing attitudes of selected members of the 
educational milieu toward evaluation*//^ 

1.3 Becoming thoroughly aware of the Individual's or group's 
unique characteristics// 

1»4 Establishing trust and rapport with all members of the 

educational milieu, but particularly with participants of 
an evaluation effort//* 

1.5 Identifying available neoteric evaluation strategies and 
techniques 

1.6 Making select members of the educational milieu aware of the 
discrepancies between the observed and the Intended// 

1. 7 Demonstrating, In other ways, the efficacy of evaluation 









^Henceforth, this symbol (*) will be used to represent a correspondence 



of some degree between task descriptions within this paper and tasks as 
described by Owens. (See Owens, T.R. , "Suggested Tasks and Roles of 
Evaluation Specialists In Education." Educational Tec hnology. 1968, 
VIII, (p. 4 - 10) 



Henceforth, this symbol (//) will be used to represent a correspondence 
of some degree between task descriptions within this paper and tasks as 
described by Bates, et al (See Bates, D. , Buser, R.L., Ellis, J. , and 
Rice, D. , "The Evaluator Development Program." Cooperative Educational 
Research Laboratory, Inc., //lO, 1967). 
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j-»8 Identifying and reducing any inhibitions toward evaluation//* 

9 Instilling within the educational community an awareness of 
•the need for evaluation// 

1» 10 Instilling within the educational community a demand for 
evaluation 

1» 11 Reinforcing positive attitudes toward evaluation 

1»12 Stating clearly the purposes of an evaluation and the role of 
the evaluator* 

1»13 Indicating clearly, when appropriate, the types of Information 
that will result from an evaluation and to whom it will be 
made available 

1»1A Establishing clearly operational procedures and mutual rules 
of ’’etiquette" with participants of an evaluation effort 

1»15 Maintaining open communication channels among the participants 
of an evaluation effort 

1»16 Thanking the participants of an evaluation effort for their 
cooperation 

1» 17 Crediting, when appropriate, selected participants for their 
cooperation and aid in bringing an evaluation effort to a 
conclusion 

' 1.18 Providing, when appropriate, feedback related to the outcomes 
of an evaluation effort to participants of the evaluation 

1» 19 Inviting and encouraging select members of the educational 
milieu to participate actively in evaluation efforts* 

1* 20 Supporting the efforts of persons Initiating and/or engaging 
in evaluation efforts 

1» 21 Consulting with and supporting the efforts of professionals 
and para-professionals who are attempting to develop a 
supportive climate for evaluation 
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2. FOCUSING AN EVALUATION; I SELECTING DECISION SITUATIONS 



As a rule, EM members will not themselves seek to initiate formal 
evaluation efforts unless (1) they are fully cognizant of the potential 
or current existence of a "problem" that must or should be resolved; and 
(2) they are convinced that there is a need to systematically collect 
additional data and/or information bearing on the problem. They will, 
of course, continue to apply grades and initiate evaluation efforts as 
a response to "institutional press" described by Taylor^ as "societal 
and professional pressures," i.e., charges from the School Board or 
legislation requiring evaluation. 

With the possible exception of an evaluation specifically designed to 
reveal the existence of unrealized problems, evaluation planning cannot, 
under normal circumstances, meaningfully begin unless a need — a problem — 
has been noted and until select members of the EM have identified, 
evaluated, and selected specific "decision situations" as the foci of the 
evaluation effort. Smith® has defined decision situations as "A situation 
in which a judgment must be made about a course of action." The function- 
ing textbook committee has entered into a decision situation, for 
example. Having been alerted by a colleague of the need to replace 
worn texts, they must now decide to reorder the same or to order a 
newer text. How they will arrive at a decision should concern us all. 

Will they rely solely upon propaganda, authority, tradition, or intui- 
tion? Or, will they systematically collect data and/or information about 
the two texts? The latter course of action certainly would be preferable. 

Not every member of the EM is sufficiently experienced or skilled to 
recognize the existence of a problem that requires the decision-maker to 
develop a solution. Others, dreading the decision-making process itself, 
choose to ignore untoward indicators. Still others may respond inflexibly 
to a problem that is only partially understood. Thus, in addition to 
assisting in the identification, evaluation, and selection of decision 
situations, evaluation specialists also must assume responsibility for 
training select members of the EM to be alert to educational needs — to 
become more sensitive to "decision stimuli," i.e., those stimuli (worn 
textbooks) which direct the attention of the decision-maker, as well as 
significant others, to a decision situation. Tasks in this Category 
include: 

2.1 Training select members of the educational milieu to become 
sensitive to. decision stimuli (including "institutional 
presses") 

2.2 Responding to decision stimuli (including "institutional presses") 

2.3 Reinforcing those individuals sensitive to decision stimuli 



^Taylor, P.A., "A Theoretical Evaluation Model." Educational and 
Psychological Measurement . 1967, 27, p. 305 - 321 

^Smith, G.R., "An Analysis of Research on Decision Situations and 

Processes." Paper presented at the Meeting of the American Educational 
Research Association, February 1967, p. 3 
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2,4 Identifying decision situations of interest and importance 
to select members of the educational milieu 



2,5 Determining the desirability of collecting evaluative infor- 
mation for each of the identified decision situations 



2,6 Identifying the major level (s) of decision-making and of 
decision-makers to be served* 



2,7 Identifying the decision-making process as it operates in a 
given setting// 



2,8 Determining meaningfulness and feasibility of collecting 
evaluative information for each of the identified decision 
situations 



2,9 Establishing priorities for the collection of evaluative infor- 
mation 



2,10 Selecting tentative, desirable, meaningful, and feasible 
decision situations for evaluation 



3. FOCUSING AN EVALUATION: II SELECTING EVALUATION STRATEGIES AND 



DEVELOPING EVALUATION PLANS 



Despite their obvious kinship, there is no denying that educational 
evaluation significantly differs from its pedagogically-oriented counter- 
part — educational research. Each tends to emphasize divergent educational 
concerns, for example. It is the intention of all educational evaluation 
to provide decision-makers (either present or future) with "practical" 
information related to the "worth" of educational experiences, outcomes, 
and artifacts. Most of the time the evaluation will be concerned with 
describing or judging "what" it is that is happening. At other times, 
the evaluation will be concerned with "why" a thing happens. 



Recently, Stake^ suggested that evaluators "Have a fundamental choice: to 
be scientific, to generalize, to evaluate to find out why ; or to be des- 
criptive, to be delimited, and to evaluate to find out what . " (p. 41) 



In contrast, one finds that it is not always the intention of educational 
research to provide outcomes which have practical educational value; some 
educational research (as it should be) is purely "theoretical." Further- 
more, educational research is only mildly interested in describing "what" 
is happening; its major concern is to discover and confirm the causes and 
effects — the "why'4 — of educational experiences and outcomes. Of the two, 
evaluation is the more flexible and readily adapts, in its ascientific 
form, to the natural chaos of ongoing action programs. Basically a 



^Stake, R.E., "Generalizability of Program Evaluation: The Need 

for Limits," Educational Product Report , 1969, 2, p. 39-41 



• 5 •• 









’’laboratory" science educational research is more restricted than evaluation 
and requires highly controlled conditions. 



What kind of evaluation strategy should the evaluator utilize? Should he 
employ what Campbell and Stanley® have called ’’quasl-experimental" research 
designs, or should he use the more traditional "experimental" design? As a 
matter of fact, the evaluator is in a position to choose "pre-experimental" 
designs. The choice, according to Stake^, "Depends on how much and in what 
directions the findings are expected to generalize, to be relevant to 
programs other than the one observed." 



Another aspect of evaluation strategy concerns the categories of infonnation 
which should be gathered as part of the evaluation. Evaluation models 
developed by Stake^® and by Stufflebeam^^ provide us with some answers to 
this problem. 

After an evaluation strategy has been selected, an evaluation plan must be 
developed. One must distinguish between evaluation plans and managerial 
plans. Evaluation plans are concerned with both general design Issues 
and the detailed accounting of the proposed content--the "what"— of the 
evaluation. Managerial plans also are concerned with specifics — the 
detailed "how" of the evaluation. 



Tasks associated with this category are: 

3.1 Establishing criteria for decision-making regarding expected 
outcomes// 

3.2 Projecting the decision situations to be served in terms of 
their locus, focus, criticality, timing, and composition of 
alternatives* 

3.3 Becoming knowledgable about relevant aspects of settings, 
conditions, and/or contexts within which the evaluation 
will occur# 



^Campbell, D.T. and Stanley, J.C., "Experimental and Quasi-experiTnental 
Designs for Research on Teaching." In Gage, N.L. (Ed), Handbook of 
Research on Teaching . Chicago: Rand McNally & Co,, 1963 

^Stake, R.E., loc. cit . 

^^Stake, R.E., "The Countenance of Educational Evaluation.’’, p. 523 - 540 

^^Stuf flebeam, D.L., "Toward a Science of Educational Evaluation," 
Educational Technology, 1968, VIII, p. 5-12 
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3,4 Projecting the decision situations to be served in terms of 
political, social, institutional, and situational constraints 

3»5 Defining policies and limits within which the evaluation must 
operate 

3.6 Making explicit and clarifying project assumptions* 

3.7 Requesting pertinent information from surveillance specialists 
and retrieval centers 

3.8 Reviewing research literature concerning similar projects in 
order to: (1) verify assumptions, (2) uncover sources of 
possible incidental gains or unwanted side effects, and (3) 
make as uniform as possible the use of tests and testing 
procedures*// 

3.9 Visiting pertinent persons and places to acquire updated 
information 

3.10 Developing the rationale and objectives for the decision 
situations 

3.11 Identifying or formulating the basic question and/or hypothesis 
of the evaluation// 

3.12 Establishing premises which will guide the evaluation* 

,ing 

3.13 Determii?/ if the evaluation goals are formative, summative, 
or both 

3.14 Determining the level of generalization for the evaluation 

3.15 Identifying, when appropriate, the pre-experimental, experi- 
mental, or quasl-experlmental research designs to be used In 
the evaluation// 

3.16 Identifying available evaluation strategies 

3.17 Selecting an appropriate evaluation strategy// 

3.18 Utilizing, when necessary, the services of evaluation specialists 
to select evaluation strategies and develop evaluation plans 



3.19 Developing, if necessary, evaluation strategies// 



3.20 Identifying segments of the educational community to be 
affected by the Intended transactions and outcomes 



3.21 Detailing, when appropriate. Intended antecedents, transactions, 
and outcomes// 
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3.22 Re-structuring, when necessary, intentions (objectives) into 
observable (measurable or describable) outcomes* 

3.23 Listing, if appropriate, the potential existence of contin- 
gencies between antecedents, transactions, and outcomes 

3.24 Listing, if appropriate, the potential congruence between 
intants and observations 

3.25 Making explicit and clarifying standards for use in the 
judgment of alternatives* 

3.26 Identifying and clarifying judgments required of the 
evaluator 

3.27 Identifying segments of the educational milieu from which 
judgments will be collected 

3.28 Estimating, when appropriate, which data-gathering techniques 
will be utilized in the collection of observables 

3.29 Identifying the sample// 

3.30 Relating the project or program to other efforts or exper- 
iences of others who have coped with similar or related 
areas, and showing how the project utilizes, builds upon, 
extends, revises, or adapts to existing knowledge// 

3.31 Coordinating efforts to develop evaluation plans 

3.32 Smoothing the efforts of others to develop evaluation plans 
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A. ORGANI25ING THE EVALUATION AND DEVELOPING MANAGERIAL PLANS 



The recent incursion o£ network-based managerial systems into the 
social sciences (see the work of Desmond Cook,^^ for example) now 
makes it meaningful to create specialized plans— to distinguish 
between evaluation and managerial plans* Evaluation plans are 
designed, partially, to focus the evaluation— to sharply define 
what will be done to answer what basic questions* In contrast, 
managerial plans are concerned mainly with the systematic organ- 
ization of evaluation components— a detailed description concerned 
with (1) how the evaluation will take place, step by step, and 
(2) what resources (money, manpower, facilities, machines) will 
be required over what period of time. It Is a complex step 
whereby a multitude of divergent components are arranged Into a 
coordinated program or project. Also, during this stage It is 
often appropriate, at the very least, to develop plans for the 
(1) selection, modification, or development of data-gatherlng tech- 
niques; (2) sampling of populations; (3) collecting of data (inc- 
luding procedures and data formats); (4) preparation of data; (5) 
treatment of data; (6) interpretation and judgment of outcomes; (7) 
reporting of outcomes; (8) storage and dissemination of reports; and 
(9) utilization of information by decision-makers. Of course not 
every evaluation effort will be concerned with developing every one of 
these nine plans, but even that must be determined. 

Management plans or systems based on the concepts of network planning 
and critical path analysis are known by a wide variety of abbreviated 
names. For example, there are PERT (Program Evaluation and Review 
Technique) and CPM (Critical Path Method). Such systems can allow for 
the planning, scheduling, and controlling of complex interrelated 
evaluation efforts and of the resources (time, money, manpower, fac- 
ilities, machines) required to execute those activities. Archibald 
and Vlllorla^^ have written an excellent technical book on management 
systems. 

Tasks appropriate for this function are; 

4.1 Reviewing and becoming familiar with the objectives of the 
evaluation effort 

4.2 Identifying those program and/or project components which 
must be completed in order to achieve the evaluation objec- 
tives. 



12 

Cook, D., "Program Evaluation and Review Technique: Applications 
in Education." OE-12024, CoopcgaftJf»e Research , Monograph No. 17, 1968. 

13 

Archibald, R.D. and Vlllorla, R.L. , Network-Based Management 
Systems (PERT/CPM) , New York: John Wiley & Sons, 1967. 
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4.6 



4.7 

4.8 

4.9 

4.10 

4.11 

4.12 

4.13 

4.14 

4.15 

4.16 

4.17 

4.18 

4.19 

4.20 

4.21 

4.22 

4.23 

4.24 
4.'25 









Reviewing and becoming familiar with Issues of criticality 
and timing* 

Identifying "milestones” and developing a planning structure 
for the program or project. 

Developing planning structures for individual projects or 
components 

Developing plans for selecting or developing data-gathering 
techniques 

Developing sampling plans 
Developing plans for coding data 
Developing plans for collecting data// 

Developing plans for preparing raw data# 

Developing plans for treating data 

Developing plans for interpreting and judging outcomes 
Developing plans for reporting data 

Developing plans for storing and disseminating information 



Developing plans for Information utilization 

Determining the dependency relationship existing among 
the Identified project components 

Arranging the componeiits of the evaluation in a network 
according to a plan 

Determining how long it will take to complete the program 
plan 

Determining the amount and type of resources required to 
complete the plan 

Modifying the plans according to known restraints 
Determining local resources available for the evaluation# 
Defining the manpower and material needs for the evaluation*# 
Constructing an evaluation budget for the project*# 
Coordinating efforts to develop managerial plans 
Smoothing efforts to develop managerial plans 
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5. ASSESSING AND MODIFYING EVALUATION AND MANAGERIAL PLANS 

As evaluation and managerial plans are being developed, they should be 
systematically assessed and final assessment should be made before they 
are submitted for funding. Flans assessments should Include stylistic 
criteria, l.e,, general communicability. They also should Include content 
criteria, l.e., legality and practicality. In a relatively recent article, 
Caldwell^^ described a plans assessment framework and suggested criteria 
by. which plans may be assessed. 



Tasks which should be Included with this factor are: 



5.1 Reviewing evaluation and managerial plans for communication 
value, l.e., are the plans clear and seductive 

5.2 Determining If the scope of the evaluation has been stated 
explicitly 

5.3 Determining when necessary. If the format of the proposed plans 
Is appropriate for the receiving agency// 

5.4 Learning the project rationale, objectives, and operational 
procedures* 

5.5 Determining the audiences, the decision-makers to be served, 
and the nature of the Implementing agency* 

5.6 Using effectively subject area or technical specialists when- 
ever necessary to review the evaluation and managerial plans// 

5.7 Determining the relevance of the proposed evaluation plan to the 
Identified decision sltuatlonG^^ 



5.8 Determining the relevance of the proposed managerial plan to the 
proposed evaluation plan 

5.9 Determining the legal status of the proposed evaluation and 
managerial plans relative to the context within which they are 
to be lmplemented@ 



5.10 Determining the congruence of the evaluation and managerial 

plans with the value systems of the context within which they 
are to be ImplementedG 



^^Caldwell, M.S., "An Approach to the Assessment of Educational 
Planning." Educational Technology, 1968, VIII, p. 5-12. 



15. 



Henceforth, this symbol (@) will be used to represent a cor- 
respondence of some degree between task descriptions within this paper 
and tasks as described by Caldwell (Ibid.) 
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5.11 Determining if the evaluation plan is within the purview of the 
agency charged with the implementation0 

5.12 Determining the compatibility of the evaluation and managerial 
plans with the value system(s) , l.e., purposes and goals of the 
implementing agnecy@ 

5.13 Determining the Impact of the evaluation and managerial plans on 
other components (sub-systems) of the system and on the weights 
and Interrelationships of these system elements@ 

5.14 Determining the practicality of the evaluation and managerial 
plans in terms of achieving its stated purposed (end-products)® 

5.15 Determining the relative desirability of the evaluation and 
managerial plans (In comparison with other plans) in terms of 
the ratio of necessary liiputs (costs) to expected outputs 
(effectiveness)® 

5.16 Consulting with clients in order to review and/or modify 
evaluation and managerial plans 



I 






I 
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5.17 Modifying the evaluation and managerial plans in terms of 
the outcomes of the assessment 



5.18 Coordinating assessment efforts 



5.19 Smoothing assessment efforts 



5.20 Submitting the proposed plans for approval and funding 
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6. SELECTING. MODIFYING, OR DEVELOPING DATA-GATHERING TECHNIQUES 



Both the educational evaluator and the educational researcher share an 
interest in so-called ’’objective" information. In addition to objective 
information", however, the evaluator is equally as intrigued by subjective 
information". As Stake^^ has explained, "An evaluation of a school 
program should portray the merit and fault perceived by well-identified 
groups, systematically gathered and processed. Thus, judgment data and 
descriptive data are both essential to the evaluation of educational 
programs." Thus, the evaluators' interest in all forms of data (be they 
’"soft" or "hard") portrays his commitment to describe fully the coun- 
tenance" of an educational experience. 



As one would predict, the evaluator's sweeping data interests are amply 
reflected in the eclectic assortment of data-gathering techniques that 
he habitually employs in his studies. These techniques range 
standardized psychometric tests through unsystematized personal observan s. 
For this reason, we have inserted the term ’’techniques for the term 
"instruments". Since the evaluator is free to «se direct personal 
observations as a means of gathering data, he can subjectively descr — e 
persons, places, things, and processes which could not otherwise be so 
easily described. He need not (though often does) rely upon replicable 
procedures. Often, the acceptance of the evaluator s description is based 
upon conviction that the evaluator is astute enough to understand the 

environment . 



To better describe outcomes, the evaluator recently has become interested 
in "criterion-referenced” tests. A relevant bibliography is presented by 
Baker 17 in a recent view. 



Tasks considered important for selecting, modifying, or developing data 
gathering techniques include: 



6.1 Reviewing and becoming familiar with the intents of an 
evaluation study 



6.2 Stating the purposes for which data-gathering techniques are 
to be used* 



6.3 Stating explicitly the objectives for which data-gathering 
techniques are to be used// 



6.4 Utilizing, when necessary, the services of test and measurement 
specialists 



6.5 Reviewing resource materials related to similar projects to un- 
cover suitable data-gathering techniques// 



l^Stake, R.E., "The Countenance of Educational Evaluation , p. 527 



17Baker, R.L., "Curriculum Evaluation", Review of Educational 
Research. 1969, 39, p. 339 - 354 
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6.6 



Securing and maintaining copies of data-gatherlng techniques// 




I 



6.7 Utilizing, when necessary, the services of surveillance 
specialists 

6.8 Developing criteria for selecting the most suitable available 
data~gatherlng techniques’^// 



6.9 Identifying pertinent techniques for which evidence indicates 
effectiveness// 
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6.10 Selecting suitable data-gatherlng techniques 

6.11 Modifying, field-testing, and revising identified data- 
gatherlng techniques// 

6.12 Developing specifications for constructing data-gatherlng 
techniques if no existing data-gatherlng techniques are 
appropriate* 

6.13 Developing, pilot testing, and revising new data-gatherlng 
techniques*// 

6.14 Collecting, when appropriate, reliability and validity 
information// 

6.15 Collecting, when necessary, normative information for use as 
standards 

6.16 Preparing administrative and scoring manuals for the newly 
modified or developed data-gatherlng techniques 

6.17 Training personnel in the administration of various data- 
gatherlng techniques 

6.18 Consulting with clients regarding available data-gatherlng 
techniques 

6.19 Advising clients regarding the development, validation, and 
normlng of various data-gathering techniques 

6.20 Aiding select members of the educational milieu in the applica- 
tion of sound tests and measurement policies, programs, and 
practices// 

6.21 Persuading others to cooperate in the development, validation, 
and normlng of specific data-gatherlng techniques 

6.22 Obtaining resources and resource personnel to facilitate the 
development, validation, and normlng of specific data- 
gatherlng techniques 
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6.23 Coordinating efforts to select, develop, modify, validate, and/ 
or norm data-gathering techniques 

6.24 Smoothing the efforts of others to select, modify, develop, 
validate, or norm data-gathering techniques 

6.25 Researching new data-gathering techniques 



7. COLLECTING DATA 

The implementation of evaluation plans often begins with the collection 
of data. This is frequently true even if a "pre-test" is not administered, 
for the evaluator is concerned with describing transactions as they take 
place. As a method, data collection is not independent of the tech- 
niques used to gather the data. The administration of some data-gathering 
techniques requires the examiner to be highly skilled and experienced. 
Indeed, the administration of some techniques requires great sensitivity — 
an "artistic" talent. But there are many other techniques which do not 
require such skill. For example, many techniques are daily administered by 
educators who are less well-trained than professional examiners. Because 
of this, one can anticipate that parents (trained as para-professional 
evaluation aides — a role not unlike the teacher's aide) might assist the 
educator during an evaluation by administering data-gathering techniques. 

Tasks related to the collection of data are: 

7.1 Specifying information needs clearly and concisely// 

7.2 Identifying information sources (populations and individuals) for 
the collection of data* 

7.3 Identifying information environments for collecting data 

7.4 Specifying methods to be used in collecting data* 

7.5 Specifying sampling procedures*// 

7.6 Specifying the schedule for data collection* 

7 . 7 Reviewing the sampling plan and schedule with relevant 
others for appropriateness and congruence with other on- 
going programs 

7.8 Utilizing, when necessary, the services of data collection 
specialists 

7.9 Preparing the sample population and relevant others for data 
collection 

7.10 Field-testing data collection methods 

7.11 Training personnel to collect and record data* 

7.12 Informing personnel of the rules of "etiquette" for collecting 
data 
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7.13 Administering evaluative data-gathering techniques and 
recording the data* 

7.14 Reviewing and modifying, when necessary, data collecting 
plans and methods 
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7.15 Obtaining resources and resource personnel for data col- 
lection efforts 

7.16 Persuading others to cooperate in the collection and 
recording of data 

7.17 Coordinating efforts to collect data 

7.18 Smoothing the efforts of others to collect and record data 

7.19 Researching data collection techniques | 

8. DATA PROCESSING: I PREPARING "RAW" DATA 
— 

In examining the evaluation process, three major types of processing 
activities — two categories of "data" processing and one category of 
"information" processing can be identified. By data processing we shall 
mean, (1) the systematic transformation and organization (the preparation) 
of "raw" data into forms suitable for the application of analytical 
operations and (2) the systematic treatment of transformed and organized 
data during the actual analytical process. This latter set of activities 
Includes the use of computers and computer-related equipment. Information 
processing refers to those activities related to the preparation of already 
prepared, treated (analyzed). Interpreted, judged, and reported data and 
outcomes for purposes of dissemination. 

Under ordinary evaluation conditions one seldom finds freshly collected 
data, that Is, raw data or unprocessed responses, to be In a form suit- 
able enough to be analyzed without first being specially prepared. The 
raw data processing function mainly entails (1) the conversion of unpro- 
cessed responses into scores or categories; (2) the transformation of scored 
responses Into data storage cards or tabulation sheets; (4) a systematic 
organizing and re-organlzlng of coded responses into forms consistent with 
Intended analytical operations; and (5) the storage and retrieval of data. 

Tasks adjunctive to the above Include the operation of scoring machinery 
and computer-related equipment. Tasks which would fall within. the domain 
of raw data processing Include: 

8.1 Providing specifications for the scoring and/or classification 
of data 






8.2 Becoming familiar with the data, the Intended analytic process, 
and available computer programs 
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8.3 Providing formats for coding data which are compatible with 
available computer programs or analytic procedures* 

8.4 Training personnel in response interpretation 

8.5 Training personnel to operate mechanical scoring units 

8.6 Scoring and/or classifying responses obtained from the admin- 
istration of data-gathering techniques* 

8.7 Utilizing, when necessary, the services of professional scoring 
services 

8.8 Transcribing memorial or taped data when: necessary 

8.9 Training personnel to operate machines related to the pre- 
paration of computer data cards 

8.10 Transferring raw data onto computer cards, tabulation sheets, 
or other data storage systems 

8.11 Utilizing, when necessary, the services of data processing 
assistants 

8.12 Providing for data storage, management and retrieval 

8.13 Utilizing, when necessary, the services of data processing 
specialists, and test and measurement specialists 

8.14 Informing relevant members of the educational milieu 
regarding the types of raw data available for heuristic 

purposes// 

8.15 Coordinating data processing activities* 

8.16 Disseminating raw data upon the request of appropriate officials 

8.17 Consulting with clients regarding the preparation of raw data 

8.18 Obtaining resources and resource personnel for preparing raw 
data 

8.19 Smoothing the efforts of others to prepare raw data 



9. DATA PROCESSING; II TREATING DATA (ANALYSIS) 

Fred Kerlinger^® defines "analysis" as "The ordering, the breaking down of 



^^Ker linger, F,N,, Foundations of Behavioral Research , New York; 
Holt, Rinehart and Winston, Inc,, 1965, p, 603 
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data Into constituent parts in order to obtain answers to. .. [evaluation] 
questions." He explains that| "The analysis of . . . [evaluation] data, 
however does not in and of itself provide the answers to ...[evaluation] 
questions. Interpretation of the data is still necessary. "i^ Thus, data 
analysis is a process during which prepared data are systematically treated 
for the purposes of description, interpretation, and the rendering of 
judgments. 

During this process the evaluator or the statistical specialist or the 
data programming specialist systematically manipulates prepared data 
according to operations dictated by statistical formulae and the language 
of the computer. Indeed, in the last decade computer science has progressed 
so rapidly that the accomplished statistical specialist is one who has 
become well acquainted with computer programming — a marriage seemingly 
enhancing both disciplines. 

Tasks which can be associated with this function are: 

9.1 Reviewing the objectives of the evaluation 

9.2 Determining the level of sophistication required by decision- 
makers 

9.3 Reviewing the evaluation design actually employed 

9.4 Reviewing the sampling procedures actually employed 

9.5 Determining the nature of the data collected 

9.6 Determining desired levels of statistical precision 

9.7 Reviewing the research literature for new statistical procedures 

9.8 Utilizing, when necessary, the services of statistical 
specialists 

9.9 Becoming familiar with available computer programs 

9.10 Selecting the analytical procedures* 

9.11 Using existing computer programs// 

9.12 Writing new computer programs when necessary* 

9.13 Utilizing, when necessary, the services of program writers 

9.14 Designating a means for performing the analysis* 

9.15 Developing a library of cookbook— like statistical forms 
and procedures 
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9.16 

9.17 

9.18 

9.19 

9.20 

9.21 

9.22 

9.23 

9.24 

9.25 

9.26 

9.27 

9.28 

9.29 



Performing the statistical computations* 

Treating data to test hypothesis determining relationships 
and/or to answer basic questions 

Obtaining > resources and resource personnel to facilitate 
the analytical process 

Utilizing, when necessary, the services of computer and computer- 
related equipment operators 

Coordinating efforts to treat the data 

Smoothing the efforts of others to treat data 

Consulting with clients regarding statistical procedures 

Researching new analytical procedures 

Producing computational documentation when appropriate// 

Advising clients regarding available computer programs 

Consulting with clients regarding computer programming and 
operations 

Maintaining a library of computer programs and their descriptions 

Determining the types of computer programs which will be 
demanded in the future 

Disseminating information pertaining to the acquisition' and/ ^ 

or modification of computer programs | 
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10. INTE RPRETING AND JUDGING OUTCOMES I 
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The evaluator who "interprets" the outcomes of an evaluation explores them 
for their meaning and implications. Interpretation as a process takes the 
treated data and makes inferences regarding various aspects of the evalua- 
tion. The evaluator might, for example, make Inferences about the "con- 
gruence" as found to exist between intentions and observations. According 
to Stake, "The data. . .are congruent if what was intended actually happens. 
To be fully congruent the Intended antecedents, transactions, and outcomes 
would have to come to pass. (This seldom happens— and often should not.)"^^ 



The evaluator also may attempt to draw conclusions about "contingencies" 
existing among antecedents, transactions, and outcomes. Stake distinguishes 
between "logical contingencies" and "empirical contingencies"; ^ 

"Whenever intents are evaluated the contingency 
criterion is one of logic. To test the logic of 
an educational contingency the evaluators rely on 
previous experience, perhaps, or research experience, 
with similar observables. No immediate observation 
of these variables, however, is necessary to test 
the strength of the contingencies among intents. 

"Evaluation of observation contingencies depends on 
empirical evidence. To say, 'this arithmetic class 
progressed rapidly because the teacher was somewhat, 
but not too sophisticated in mathematics' demands 
empirical data, either from within the evaluation or 
from the research literature. "21 



Interpretation is but one concern of the evaluator; he must also indicate 
the perceived merits and shortcomings of whatever it was that he evaluated. 
Stake and Denny22 claim that : 

"Evaluation is not a search for cause and effect 
(as is research) , an inventory of present status , 
or a prediction of future success. It is something 
of all of these but only as they contribute to 
understanding substance, function, and worth." 



20 

Stake, R.E., loc. cit . 

^ ^Ibid . 

22 

Stake, R. E. and Denny, T. , "Needed Concepts and Techniques for 
Utilizing more fully the Potential of Evaluation." In the Sixty-eighth 
Yearbook of the National Society for the Study of Education; Educational 
Evaluation; New Roles. New Means . Chicago; University of Chicago Press, 
p. 370 
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Scriven^^ also has a similar concept of evaluation J 

“Evaluation is itself a methodological activity which 
is essentially similar whether we are trying to evaluate 
coffee machines or teaching machines, plans for a house 
or plans for a curriculum* The activity consists simply 
in the gathering and combining of performance data with 
a weighted set of goal scales to yield either comparative 
or numerical ratings, and in the justification of (a) the 
data - gathering instruments, (b) the weightings, and (c) 
the selection of goals." 

Finally, the evaluator is urged to make recommendations. 

Tasks related to the interpretation and judgment of outcomes are; 

10.1 Reviewing the objectives of the evaluation 

1Q« 2 Becoming thoroughly familiar with the transactions of 

the evaluation j the basic questions asked; the criteria, 
model, procedure, and techniques employed; and the data 
collected# 

10.3 Verifying the statistical analysis and the procedures 
used to collect and process the data 
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10.4 Comparing observed antecedents, transactions and out- 1 

comes with Intended antecedents, transactions, and | 

outcomes# I 

f' 

10.5 Describing points of congruence and Incongruence# ] 

I 

10.6 Making Inferences about contingencies among transac- i 

tlons and outcomes I 



10.7 Interpreting the resu3Lts of the evaluation program in 
terms of given criteria* 

10.8 Establishing relationships and distinguishing between 
those outcomes that result from treatment application 
and those contingent upon antecedent conditions# 

10.9 Utilizing, when necessary, the services of evaluation, 
statistical, and subject area specialists in order to 
Interpret the data 
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Scrlven, M. S., "The Methodology of Evaluation." In AERA Monograph 
Series on Curriculum Evaluation; 1. Perspectives on Curriculum 
Evaluation . Chicago; Rand McNally, 1967, p. 39-83 
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10«10 Determining if any unintended outcomes occurred// 

10.11 Identifying the ways in which segments of the educational 
community, i.e», the institution, its staff, students, and 
community, etc. were affected by the observed transactions 
and outcomes 

10.12 Comparing generalizations from the literature and theory 
with those drawn from the evaluation outcomes// 

10.13 Reviewing the judgments required of the evaluator 

10.14 Obtaining judgments concerning the outcomes of the evalua- 
tion from relevant members of the educational milieu 



I 10.15 Judging the "worth” of the outcomes of the evaluation 

I . " “ 

[ 10.16 Rendering judgments regarding the worth of alternative 

I strategies as employed in the evaluation# 

I 

10.17 Rendering judgments as to the significance of the ob- 

I served transactions for various segments of the educa- 

tional milieu# 

f 10.18 Judging if the unintentional outcomes are unwanted side 

effects or incidental gains# 

•• 10.19 Rendering judgments regarding the worth and relevance of 

data-gathering techniques as used in the evaluation# 

{ 

10.20 Rendering judgments as to the overall quality of the 
evaluation effort 

10.21 Recommending, if appropriate, future modifications for 
the data-gathering techniques 

10.22 Identifying and recommending alternative strategies# 

10.23 Identifying and recommending procedures to control or 
reduce unwanted side effects# 



10.24 Identifying and recommending procedures to control or 
enhance incidental gains 

10.25 Providing counsel to relevant members of the educational 
milieu regarding the interpretations and implications of 
the judgments rendered# 

10.26 Consulting with clients regarding interpretation and 
judging outcomes 

10.27 Obtaining resources and resource personnel to facilitate 
the interpreting and judging of the outcomes 

10.28 Coordinating efforts to interpret and judge outcomes 



[ 10.29 Smoothing efforts to interpret and judge outcomes 

I 




11. REPORTING OUTCOMES 



Reporting evaluation outcomes requires an unusually high degree of tech- 
nical skill and evaluation "savvy. " Stake and Denny^^ emphasize this 
point j 

"Evaluation reports, unfortunately, usually tell 
little more than that the work proposed was com- 
pleted, that the complaints of the staff were 
justified, and that there were greater differences 
within groups of students (or schools or curriculums) 
than there were between the groups. Most evaluation 
reports give only the participants some notion of 
what occurred; the outsider gains little insight. 

Most formal reports avoid explicit subjective judge- 
ments by insiders and outsiders as if they were evil. 

"Educators and laymen alike cannot non-visualize and 
explain what is happening in our classrooms. Part of 
the reason for this failing is our inability to share 
perceptions and measurements. Part is our lack of mo- 
tivation to share them. What should be told? What 
should be shared? Our needs are not only procedural; 
we need also a commitment to full and accurate re- 
porting." 

The reporting function is the fifth part of Stuff lebeam’s^^ evaluation 
design. Concerned less with what should be told and shared. Stuff lebeam 
stresses the procedural aspects of reporting: 

"The purpose of this part, the reporting part of a 
design, is to insure that decision-makers will have 
timely access to the information they need and that 
they will receive it in a manner and form which facili- 
tates their use of the information." 

One must accept the fact that evaluation report writing is a difficult, 
often unrewarding job. 









Tasks which would be associated with reporting the outcomes of evaluation 
are: 

11.1 Becoming familiar with available means for reporting outcomes 

11.2 Researching new methods for reporting outcomes 

11.3 Specifying the format for evaluation reports 



.24 

Stake, R.E. and Denny, T. , loc. clt . 

25 

Stufflebeam, D.L., loc.cit . 
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11 « 4 Becoming aware of the professional understandings of those 

who are to receive the evaluation results# 
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11.5 Scheduling the reporting of outcomes 

11.6 Specifying means for reporting the outcomes of the evaluation 
to relevant audiences* 

11.7 Coordinating efforts to prepare a report 

11.8 Preparing a report of the evaluation that will be understandable 
to the public. It will serve and include meaningful terms, 
tables, charts, graphs, illustrations, and answers to the basic 
questions of the evaluation# 

11. 9 Describing the intended antecedents, transactions, and outcomes 

11.10 Describing processes and procedures by which evaluative data 
were gathered and judgments rendered# 



: 11-11 

11.12 

11.13 

: - 11.14 



Describing the observed antecedents, transactions, and outcomes 
Describing criteria# 

Describing explicitly unintentional outcomes# 

Detailing rendered judgments 






11.15 Reporting the limitations of the evaluation# 

11.16 Utilizing, when necessary, the services of reporting 
specialists in order to report the outcomes 

11.17 Obtaining resources and resource personnel to facilitate 
the reporting of outcomes 

11.18 Coordinating efforts to produce a report 

11.19 Preparing findings and recommendations to the decision - 
makers in an understandable manner*# 

11.20 Obtaining the decision-makers* reactions to the report 

11. 21 Providing evaluation abstracts or summaries for presenta- 
tion to specific groups* 

11. 22 Packaging the outcomes of the evaluation for purposes of 
presentation to appropriate publics# 

11. 23 Obtaining reactions to the presentations 

11.24 Consulting with clients regarding the reporting of outcomes 
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11.25 

11.26 



Smoothing the efforts of others to prepare and produce reports 

Reporting subsequent modifications in transactions and their 
observed outcomes# 
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12, INFORMATION PROCESSING AND DISSEMINATION 

In 8., ''information processing" was described as processing already~ 
prepared) treated) interpreted) judged) and reported data for purposes 
of dissemination. Information processing activities include reviewing 
evaluation literature) collecting evaluation reports, organizing and 
I storing evaluative-based information) packaging evaluative information 

and disseminating information pertaining to the field of evaluation, 

I "Data banks" also are included in this category. 






Oddly, there are no existing agencies, large or small, that have assumed 
the responsibility for processing and disseminating all forms of evaluative 
information. Some agencies have assumed the responsibility of processing 
and disseminating specific types of information. (Of note is the work of 
Popham and Skager26.) (Their plans involve collecting, storing, processing, 
and distributing objectives and evaluation measures.) And other agencies 
have disseminated general evaluation information for the non— professional 
evaluator, for example. The combined efforts of the Cooperative Educational 
Research Laboratory, Inc. (CERLI) and the Center for Instructional Research 
and Curriculum Evaluation (CIRCE) resulted in the dissemination of a 
telescoped "evaluation kit," containing pertinent evaluation information^^. 
Clearly, some super-agency to assume the responsibility of processing and 
disseminating evaluation-related information should be established. 



.28 



Cuba has suggested the development of. 



"A National Information Center for Education (NICE), 
which would have as its purpose organizing, processing, 
and reporting evaluative information. It would develop 
and maintain mechanisms that would serve these ends. 

The agency would maintain records of the goals and 
organization of Federal and other funding programs in 
support of education, of the objectives and procedures 
of program elements within those agencies, of the 
information requirements of major audiences for educa- 
tion, and particularly the scope, sequence, timing, 
locus, focus, and criticality of decisions requiring 
evaluative information. NICE would also be responsible 
for developing operational Instruments for such data 
collection and treatment." 



26 



Popham, W.J. and Skager, R.W. , "Instructional Objectives Measurement 
System. Progress in Evaluation Study," Third Annual Report to the U.S. 
Office of Education, Center for the Study of Evaluation. Los Angeles: 
Graduate School of Education, University of California, 1968, p. 113 - 115 



27 

Educational Products Information Exchange Institute, "Evaluation Kit: 
Tools and Techniques." Educational Products Report . 1969, 2, Information 
Supplement //5 (whole) . 



28 

Cuba, E.G. , "Confronting the Problems of Educational Evaluation: A Call 
for a Consortium of Relevant Agencies." A paper presented at an Invitational 
Conference on Educational Evaluation, Washington, D.C., 1967, p. 11; 13 




25 - 


















Cuba is Indeed anticipating a system which will be welcomed by all 
members of the EM, and especially by evaluators. However, until such 
an agency is developed, various components of EEFCS must take on some 
of the activities described above. 

Tasks which seem to be associated with this category are; 

H*}- Providing specifications for the coding and cataloguing 
of evaluative information and processed data 

12^ Training personnel in information processing and dissemination 

^ Detailing the scope of information processing and dissemination 
activities performed by the given agency 

12^ Identifying potential sources of evaluative information and 
processed data 

12^ Requesting evaluation information and processed data from 
likely sources 

^2., 6 Collecting evaluative information and processed data 

1^*7 Visiting pertinent persons and places to acquire updated 
evaluation Information 



1^*8 Utilizing, if necessary, the services of surveillance and 
subject area specialists 

12^ Conducting conferences regarding specific evaluation-related 
issues of current importance 



^2,10 Requesting select members of the EM to> prepare reviews of 
specific evaluation areas 



Coding and cataloguing evaluation Information and processed 
data 



^2., 12 Utilizing the services of information processing specialists 



^2,13 Transferring evaluation information into information storage 
systems 



^2,1^ Providing for information storage, management, and retrieval 
12» 15 Implementing and maintaining a data bank*// 



12,16 Becoming familiar with available information media methods 
for reporting information 
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12.17 Developing, if necessary, new reporting styles 

12.18 Specifying the format for reporting evaluation information 

12.19 Scheduling the reporting of evaluation information 

12. 20 Reporting evaluation information in a highly communicable 
fashion 

12.21 Providing evaluation abstracts or summaries 

12.22 Utilizing, if necessary, the services of reporting specialists 

12. 23 Identifying potential recipients of evaluative information and 
processed data 

12. 24 Informing relevant members of the educational milieu regarding 
the types of evaluation information which may be requested 

12.25 Communicating to potential users regarding the availability 
and use of the data bank// 

12. 26 Packaging the information for purposes of presentation to 
appropriate publics// 

12.27 Disseminating evaluative information and processed data 

12. 28 Consulting with clients regarding the retrieval of evaluative 
information and processed data 

12.29 Coordinating evaluation information processing and dissemina- 
tion activities 

12.30 Coordinating data bank activities 

12.31 Smoothing the efforts of others to acquire specific evaluation 
information and processed data 

12.32 Obtaining resources and resource personnel to process and 
disseminate evaluation information and processed data 

12.33 Obtaining reactions from relevant members of the educational 
milieu regarding the reporting of evaluation information 

12.34 Surveying select members of the educational milieu regarding 
their evaluation information needs 

12.35 Determining future evaluation information needs 
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13. DECISION MAKING; THE UTILIZATION OF INFORMATION 

The evaluation process has now run its full cycle. We began this 
process by modifying attitudes, that is, by attempting to instill 
within the educational community a demand t o engage in formal evalua- 
tion projects. Then we trained EM members to become sensitive to 
decision stimuli so that they might later identify decision situations. 
Following this, evaluation strategies were selected and evaluation and 
managerial plans were developed. After the plans had been completed, 
they underwent a final assessment and were modified if deemed necessary. 
Data-gathering techniques were selected or developed and data was 
collected. Finally, the raw data was systematically prepared, treated, 
interpreted, judged, reported, and disseminated. 



Now a decision-maker has information, and for all intents and purposes 
the evaluation will be terminated. The "feedback" of information to a 
decision-maker — the goal of an evaluation effort — has been accomplished. 
But, the ways in which the decision-makers employ the information they 
receive is as crucial to the evaluation process as any of the preceding 
steps. If decision-makers do not use formally gathered information 
(even to the extent of informing themselves about "what" is happening), 
then the raison d’etre for evaluation efforts has vanished. An evalua- 
tion effort makes no sense if its product — information — is not utilized. 



I ’ In this respect, Wilhelms^^ suggests that: 

t 

I . "Regardless of whether the evaluation is formal 

I or informal — and equally regardless of whether 

I it is ’good’ or ’sensitive’ or ’adequate’ — it 

I has one thing in common with every other system 

I of feedback: When it has blended into the back- 

I ground system of purposes and values and policies, 

I controls the next step . This is simply a fact 

I of life; all our decisions are conditional by 

[ perceptions of how we are doing in terms of what 

I we hope to do . " 



That "next step" can be excruciatingly painful especially if the decision- 
I maker is unfamiliar with decision-making strategies. This is a likely 

i occurrence according to Guba^O^ v7ho claims that: 

** 

L 



I ^^Wilhelms, F.T., "Evaluation as Feedback." In Wilhelms, F.T. (Ed) 

I Evaluation as Feedback and Guide . Washington, D.C.: Association for 

I Supervision and Curriculum Development, NEA, 1967, p. 3 

I' 

S ^^Guba, E.G., "The Failure of Educational Evaluation," p. 35 
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"At present, no adequate knowledge of decision 
processes and associated information require- 
ments relative to educational programs exist. 

Nor is there any ongoing program to provide 
this knowledge. 

Not too long ago, Scriven"^ advised the evaluator to assume the responsi- 
bility for developing objectives. Similar advice would not be inappropriate 
now. EEFCS members should, in all due haste, seek ways to develop programs 
designed to better understand the decision processes. This is a must I For 
any reason if the decision-maker cannot utilize evaluative information, he 
will not be inclined to engage in or support future evaluations. It would 
be most tragic if efforts expended to create a supportive climate for 
evaluation come to naught. 

Tasks relative to the decision-making process are: 

13.1 Becoming familiar with the contents of an evaluation report 

13. 2 Assessing the perspicacity of the evaluation report 

13.3 Using effectively select members of the educational milieu 
to assess the perspicacity of the evaluation report 

13.4 Determining the nature of the institutions involved in the 
decision solution// 

13.5 Reviewing the major level (s) of decision-making to be 
served* 

13. 6 Reviewing the decision-making process as it operates in a 
given setting*// 

13.7 Developing taxonomies of educational decisions 

13.8 Identifying decision-making criteria 

13.9 Selecting decision-making criteria 

13.10 Identifying decision-making models 

13.11 Developing, if necessary, a decision-making model 

13.12 Selecting a decision-making model 

13.13 Identifying and proposing alternative decision solutions 
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Scriven, M. S. , loc. cit . 



13»14 Assessing the decision solution for its relevance, legality, 
congruence, relatedness, compatability, impact, practicality, 
and relative desirability 

13»15 Utilizing effectively select members of the educational 
milieu to assess the proposed decision solution 

13.16 Selecting desirable and feasible decision solutions 

13.17 Reinforcing those individuals who produce "rational" 
decision solutions 

13.18 Reinforcing those individuals who produce "creative and 
feasible" decision solutions 

13.19 Training select members of the educational milieu in 
decision-making strategies 

13.20 Inviting and encouraging select members of the educational 
milieu to participate actively in decision-making processes 

13.21 Coordinating the decision-making process 

13.22 Providing directives, guidelines, and/or other needed 
assistance to decision-makers for purposes of improving 
the usefulness of transmitted data in the decision-making 
process// 

13.23 Smoothing the efforts of decision-makers to derive and select 
decision solutions 

13 . 24 Obtaining reactions to the decision solutions from relevant 
members of the educational milieu 

ft 

13.25 Assisting the decision-maker in formulating new questions 
for future evaluation based upon the original evaluation 
findings//* 

13.26 Using evaluation findings as the basis for discussion in 
teacher in-service training sessions* 

13.27 Conducting sensitivity sessions to induce selected members 
of the educational milieu to make use of the evaluative 
information in decision solutions 

13.28 Suggesting to decision-makers techniques by which they 
themselves can evaluate and modify behavior* 
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PERFORMING ACTIVITIES AND RESEAR CH RELATED TO EVALUATION AND 
TO THE DEVELOPMENT. DIFFUSION. AND ADOPTION OF EVALUATION 



Research specifically related to evaluation is vital to the success of 
evaluation as a discipline. It is imperative that new evaluation tech- 
and strategies be developed. Referring to this latter point, 

Guba*^^ indicates that; 

"These facts lead me to the conclusion that some 
new evaluation strategy free of these (see below) 
defects is required before evaluation as a science 
can make its next major strides. Until we have de- 
veloped a theory which permits evaluation to occur 
continuously (and to provide continuous feedback of 
relevant data) , to be free of the constraints imposed 
by conventional experimental design theory, to be 
open to alterations and refinements in the treatment ■ 
during the evaluation period, and to yield results 
valid in the septic world of the classroom as well 
as in the antiseptic world of the laboratory, we 
will make little progress." 

More research is needed to determine how well our available strategies 
and techniques fare under close scrutiny. Evaluation-related variables 
of all kinds must be examined and normative information — standards — 
must be collected and processed. But, if neoteric evaluation as a 
discipline is ever to be adopted by the school community, more activity 
and additional research related to development, diffusion, and adoption 
activities (as described by Clark and Guba^^ and as amplified by Stuffle- 
beam and Westerlund^^) must be performed. If Neoteric evaluation strategies 
can be classified as innovations, they become subject to the procedures re- 
lated to and necessary for change in education. J 

Tasks related to the performance of research related to evaluation and activi— I 

ties related to the development, diffusion and adoption of evaluation are: j 

14.1 Defining explicitly the meanings of evaluation-based terms f 

14.2 Developing and testing evaluation models* I 



14.3 Performing research related to new methodological techniques* 

14.4 Drafting plans for constructing idealized evaluation strategies 
and techniques for usage in select settings of the educational 
milieu 
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14.5 



14.6 



14.7 



14.8 



14.9 



14.12 



14.13 



14.14 



14.17 



14.19 



14.20 



14.21 



14.22 



14.23 



14.24 



14.25 



14.26 



Constructing idealized evaluation strategies and techniques 
for usage in select settings of the educational milieu 



Integrating the components of the strategies and techniques 
into operating systems for usage in select settings of the 
educational milieu 



Developing procedures for creating widespread awareness of 
the neoteric evaluation strategies and techniques 



Developing situations by which individuals can examine and 
assess operating qualities of the neoteric evaluation 
strategies and techniques 



Researching procedures for the training of local personnel to 
manage, operate, service, and utilize neoteric evaluation 
strategies and techniques 



14.10 Developing situations for the trial use of the evaluation 
strategies and techniques 



14.11 Modifying the neoteric evaluation strategies and techniques 

to fit the particular circumstances of the adopting institution 



Performing studies to assure the assimilation of the evaluation 
strategies and techniques by the adopting institution 



Conducting longitudinal studies to determine effects of 
specific variables over time* 



Performing experimental research on some of the substantive 
areas being evaluated* 



14.15 Collecting standards of all kinds 

14.16 Developing a taxonomy for standards 



Performing case studies or other types of research to learn more 
about the nature of children involved in programs being evaluated// 



14.18 Conducting simulation studies and predictive studies* 



Conducting surveys related to educational needs, and uses and 
abuses of evaluation 



Determining the applicability of various data-gathering tech- 
niques for special populations 



Comparing alternative strategies and techniques for instilling 
in select members of the educational milieu an awareness of a 
need for evaluation 



Comparing alternative strategies for instilling in select mem- 
bers of the educational milieu a demand for evaluation 



Determining attitudes toward evaluation and readiness for change 
Coordinating research and activities relating to the development, 
diffusion and adoption of evaluation 

Coordinating evaluation based research in general 

Smoothing general research activities and activities related to 

development, diffusion and adoption of evaluation 
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15. ADMINISTERING AND COORDINATING ACTIVITIES IN AN EVALUATION 
FACILITATION AND COORDINATION SYSTEM 



All organizational entities require some personnel to direct and supervise 
the activities of the staff. Though not always directly pertinent to the 
evaluation process, these are vital functions. In the real world the evalua- 
tion probably would not be initiated without the aid of the administrative 
staff. 



Activities to be included within the framework of administration-type tasks 
are: 

15.1 Stating explicitly the broad purposes of an evaluation and/or 
facilitation entity# 

15.2 Developing specific policies and general guidelines for the 
operation of an evaluation and/or facilitation entity 

15.3 Identifying and assessing alternative objectives as possible 
goals for an evaluation and/or facilitation entity# 

15.4 Defining criteria for selecting objectives for an evaluation 
and/or facilitation entity# 

15.5 Selecting and assigning priorities to objectives for an 
evaluation and/or facilitation entity# 

15.6 Stimulating and assisting in periodic evaluation, reflection, 
and revision of purposes and/or objectives# 

15.7 Defining staff and resource requirements for operating an 
evaluation and/or facilitation entity* 

15.8 Developing plans to meet staff and resource requirements* 

15.9 Developing job descriptions 

15.10 Constructing, securing, and managing budgets* 

15.11 Developing policies and procedures for the selection, assignment, 
retention, dismissal, promotion, and in-service growth of per- 
sonnel# 



15.12 Establishing criteria for evaluating the on-the-job performance 
of personnel 

15.13 Developing policies and techniques for evaluating on-the-job 
performance of personnel# 



15.14 Reviewing all evaluation designs, instruments, and reports be- 
fore they are used or released for distribution* 

- 33 - 




wmmmm 























15«15 Identifying sources, i.e., foundations or agencies, which 
have indicated an interest in supporting programs or 
projects similar in kind to the submitting evaluation and/or 
facilitation entity# 

15.16 Making an informal contact with the agency to which the proposal 
will be submitted# 



15.17 

15.18 

15.19 

15.20 

15.21 

15.22 

15.23 

15.24 

15.25 

15.26 

15.27 

15.28 

15.29 

15.30 



Developing an overall managerial plan for an evaluation 
and/or facilitation entity 



Supervising the training, research, facilitation and co- 
ordination services performed by staff members* 



Arranging for in-service training of the staff* 

Organizing the tasks within the entity in order to utilize 
the unique talents of each member* 



Maintaining conditions conducive to high morale and job 
efficiency 

Arranging for an independent evaluation of the activities 
of an evaluation and/or facilitation entity* 

Coordinating evaluation, facilitation and/or coordinating 
activities within the evaluating group 

Coordinating evaluation, facilitation and or coordinating 
activities within the school 

Coordinating evaluation, facilitation and/or coordinating 
activities within the district 

Coordinating evaluation, facilitation and/or coordinating 
activities between districts 

Coordinating evaluation, facilitation and/or coordinating 
activities within a cooperative mult-district unit 

Coordinating evaluation, facilitation and/or coordinating 
activities between cooperative multi-district units 

Coordinating evaluation, facilitation and/or coordinating 
activities within the state 



Smoothing the administrating and coordinating efforts of 
others 
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16 . PROVIDING FACILITATION SERVICES 



It would seem logical to consider evaluation a "service-oriented pro- 
fession not unlike that of law and medicine whose practitioners provide 
information and formulate decisions for their clients. In one respect, 
however, evaluation differs: the professional evaluator’s clientele con- 
sists of "amateur" evaluators who know all about or at least express 
opinions about educational practices, institutions, problems, etc. 



Nevertheless, in this age of specialization, some select evaluators 
should function as facilitators of the decision-making process. Pro- 
fessional (highly qualified) evaluators not only should "evaluate" but 
they should facilitate and coordinate evaluation efforts that dynamize 
the decision-making process. 



An Educational Evaluation Facilitation and Coordination System would 
provide facilitative services for evaluation and decision-making efforts. 

To accomplish its mission, the system would be designed to increase the 
quality and usage of evaluation strategies and techniques in educational 
systems by providing services to professional, para— professional, temporary 
and non-professional evaluators. 



Tasks associated with these services include: 



16.1 



Surveying the training needs of select members of 
the educational milieu// 



16.2 



Developing instructional objectives, plans, aids, and 
materials for training// 



16.3 



Training select members of the educational milieu in 
tasks associated with their roles 



16.4 Training select members of the educational milieu in tasks 
related to evaluation facilitation and/or coordination 



16.5 Obtaining from select members of the educational milieu 
reactions to training 



16.6 Surveying the service needs of select members of the 
educational milieu 



16.7 Facilitating, generally, the efforts of select members of 
the educational milieu to undertake evaluations 



16.8 



Assisting select members of the educational milieu to 
develop objectives// 



16.9 Reducing impediments to evaluations 



16.10 Developing systems of support and reinforcement to those 
individuals undertaking evaluative efforts 
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Obtaining from select members of the educational milieu 
reactions to facilitative efforts 

16^2 Facilitating, generally, the efforts of select members 
of the educational milieu to develop solutions for 
operating problems 

Obtaining resources and resource personnel for the 
facilitation of evaluation efforts, the coordination 
of evaluation efforts and for decision-making efforts 
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